UCI MAE 195: Intro to Machine Learning



1 Abstract

Classification of typical daily human activity many applicable uses, especially when the hardware reliance is a mobile phone and associated smart watch. Within the machine learning field this type of problem is known as human activity recognition (HAR). Within this project, we are interesting in seeing to the accuracy that we can attain using different machine learning methods for classes of actions. The data set of interest is the WISDM Smartphone and Smart watch Activity and Bio metrics Data set from the UCI machine learning repository. Some of the actions in the data set are more distinct and so classification from one another will be simpler than one with mere nuances separating them (different eating action vs running). From literature recommendations, three machine learning models will were used and compared along with a novel binning method for increasing accuracy. Of the three models it was evident that random forest gave the best results, but there is a discrepancy between cross validation and testing accuracies. Most actions were able to be able to be classified, peak accuracy of 69.3%, but the eating activities proved to be a challenge.

2 Intro

As part of the MAE 195 Intro to Machine Learning course at UCI, we were tasked with creating a classifier for a common HAR data set. The task was not just to create the classifier, but attempt to understand the shortcomings of the data set and models in getting to perfect classification. This page will cover how the problem was attacked and what results were found. Alongside the primary discussion we will converse about what the next steps would be and implications to real world problems.

3 Methods

Highlighted here is the approach and rational for the problem solution. To establish credibility in my claims, I will continuously be citing well-respected journals and work within the field. When possible, I will use quantitative data or more discrete representations of data or ideas to limit the possibility of grey areas. The goal is to aggregate and more importantly validate the train of thought proceeding through the work.

3.1 Code Sections

Here I will highlight the primary groups that make up the code and give a brief overview of what operations occur within each group. For the sake of brevity, most sections detailed here do not encompass the entire code. For the raw code, please see the appendices.

3.1.1 Packages Used

I will not highlight my uses for all of the packages, but I will mention some noteworthy ones as they could be useful for someone else with similar problems.

  • tictoc: Used for timing of function calls and time optimization
  • caTools: Used for train and test splitting of data
  • caret: Training harness backbone for determining the best model
  • Amelia: Helpful for finding NA’s in data
  • RWeka: Used for reading in ARFF files
  • parallel/doParallel: Multi-threading training operations for caret

3.1.2 Data Import

Originally I was importing the raw data and doing a fresh set of feature engineering, but due to the computation time and the inclusion of a precomputed feature set with the data set I opted to import that instead and move to modeling. Attempts at feature engineering are in the GitHub linked in the appendix. The data set very nicely includes ARFF files for each user broken out on the basis of phone accelerometer, phone gyroscope, watch accelerometer, and watch gyroscope [1]. This section imports each of these files and create a row that represents the features for a selection in the combination set of user, action, and sensor. The data is constructed in 10-second windows such that each of the combinations mentioned prior have 17 windows of features.

3.1.3 Feature Engineering

When considering the time series data, it is crucial that we are able to retrieve attributes of the time series to be able to characterize it. The process of generating or extracting these attributes is called feature engineering [2]. In my experience with this project, this is not trivial as the choice and fidelity of attributes can make a meaningful impact on your results. Take figure 1 where all the images have the same XY mean and standard deviation. If those were the only parameters from our feature engineering, we would not be able to classify the very distinctly different images correctly [3]. For the given data set there was an included feature set that I will be using, but this was only after attempting my own feature engineering and seeing that the features supplied could be corroborated with literature I had read [4]. Some common features that are used are spectral peaks using FFTs and autocorrelation [5]. Neither of these were used here, but they seem promising.

Representation of figures with identical basic statistics.

Representation of figures with identical basic statistics.

3.1.4 Feature Reorganization

The features imported from the ARFF files are not in a form that can be immediately sent to our models. We need to row bind pairs of 4 rows that are from the 4 sensor suites. This way we can get each row to represent all features for that window of time for a given user’s action. This is also where highly-correlated predictors are removed to reduce the computational load and reduce bias [6].

First 10 rows and features of the features dataframe.
User Action PA_X0 PA_X1 PA_X2 PA_X3 PA_X4 PA_X5 PA_X6 PA_X7
1600 A -0.0891426 1.0116878 0.5034891 -0.4131524 -0.4371668 -0.4234352 -0.2286574 -0.2252173
1600 A 0.0109645 0.9101767 0.4842371 -0.4312157 -0.4371668 -0.4234352 -0.2286574 -0.2252173
1600 A 0.1235851 0.8763397 0.3879768 -0.4854056 -0.4199875 -0.4234352 -0.2286574 -0.2252173
1600 A 0.1110717 1.0962803 0.1569521 -0.4673423 -0.4371668 -0.4234352 -0.2286574 -0.2252173
1600 A -0.1391962 0.9609322 0.6960097 -0.4673423 -0.4371668 -0.4234352 -0.2286574 -0.2252173
1600 A 0.0735315 1.0455248 0.2532124 -0.4492790 -0.4371668 -0.4234352 -0.2286574 -0.2252173
1600 A 0.1736387 1.0455248 0.0799439 -0.4312157 -0.4371668 -0.4234352 -0.2286574 -0.2252173
1600 A 0.0860449 1.2993025 -0.0163163 -0.4854056 -0.4371668 -0.4234352 -0.2286574 -0.2252173
1600 A 0.2487190 1.0962803 -0.0548205 -0.4673423 -0.4371668 -0.4234352 -0.2286574 -0.2252173
1600 A 0.1861520 1.1808729 -0.0933246 -0.4312157 -0.4371668 -0.4234352 -0.2286574 -0.2252173

3.1.5 Train/Test Split

The purpose of this section is to take the dataframe of features and split them into a test and train set. It helps to do this post feature generation as it’s very easy to implement k-fold cross-validation. If the data were to be split before feature generation, repeat operations would bog down the code execution. Given the premise of the problem, where a user would have actions that need to be characterized, it was determined that the splits would be on the basis of the dataframe users.

users <- unique(df1$User)
n_user = length(users)
inds <- sample.split(1:n_user, SplitRatio = split_ratio)
users_train <- users[inds]
users_test <- users[!inds]
df_train <- bind_rows(df_train, df[df$User %in% users_train,])
df_test <- bind_rows(df_test, df[df$User %in% users_test,])

3.1.6 Model Testing

For the data set we will evaluate 3 models: K Nearest Neighbors, Support Vector Machines, and Random Forest. These models have shown viability in literature [7]. The caret package was very helpful in developing a testing harness for these models [8]. With a control variable, we can command different models to all follow the same cross-validation and enable parallel processing of training tasks [9]. The train function call also will vary important inputs for each model and report back on what parameters are optimal. The data used for training is the train subset from before as the validation predict must be done with unseen data. Unlike a traditional train/test split, the training data here is not just used for model training but is instead used to develop an understanding of model accuracy while also training. The test data set acts as a validation set to ensure that we are getting performance similar to that specified from our cross validation.

# Run algorithms using 5-fold cross validation
control <- trainControl(method="cv", number=5, allowParallel=T)
metric <- "Accuracy"

# kNN
set.seed(100)
print('KNN Fit Start')
fit.knn <- train(Action ~ ., data=data_train, method="knn",
                 metric=metric, trControl=control)
pred.knn <- predict(fit.knn, data_test)
confusionMatrix(pred.knn, data_test$Action)

3.1.7 Binned Accuracy

As mentioned prior, one of the additions that supplements the model is binning the predictions from the classifier by action. As a reminder, the individual predictions are ~10s windows that have been feature-reduced. This means that for a given action, the 3 minute time series, we have 17 predictions for that action observation. To get a more accurate estimation, we will use the max of the binned predictions to determine the best estimate of what the action for the entire time series was. Below is an example of what the binning might look like and what final predictions were made.

4 Results

The results were very interesting as the test validation and cross validation ac curacies were very different.

4.1 Results Table

Results for the 3 models with all ac curacies.
Model Resample Accuracy Kappa Accuracy.Test Accuracy.Binned
KNN Fold1 0.9210651 0.9164214 0.5629301 0.6000000
KNN Fold2 0.9209524 0.9162990 0.5629301 0.6000000
KNN Fold5 0.9204762 0.9157942 0.5629301 0.6000000
KNN Fold4 0.9324131 0.9284361 0.5629301 0.6000000
KNN Fold3 0.9262607 0.9219206 0.5629301 0.6000000
SVM Fold1 0.9067998 0.9013163 0.6734739 0.7358491
SVM Fold4 0.9105188 0.9052542 0.6734739 0.7358491
SVM Fold3 0.9015224 0.8957301 0.6734739 0.7358491
SVM Fold2 0.9138095 0.9087398 0.6734739 0.7358491
SVM Fold5 0.9076190 0.9021820 0.6734739 0.7358491
RF Fold1 0.9158345 0.9108831 0.6925638 0.7622642
RF Fold3 0.9067555 0.9012704 0.6925638 0.7622642
RF Fold2 0.9095238 0.9042014 0.6925638 0.7622642
RF Fold5 0.9014286 0.8956290 0.6925638 0.7622642
RF Fold4 0.9062351 0.9007176 0.6925638 0.7622642

4.2 Confusion Matrices

Refer to the below for action decoding.

Action codes and their relevant actions
A B C D E F G H I J K L M O P Q R S
Walking Jogging Stairs Sitting Standing Typing Teeth Soup Chips Pasta Drinking Sandwich Kicking Catch Dribbling Writing Clapping Folding

4.2.1 Test Confusion Matrices

KNN Test Confusion
A B C D E F G H I J K L M O P Q R S
A 128 0 22 0 1 0 0 0 0 0 0 0 2 0 3 0 0 0
B 0 241 1 0 0 0 0 0 0 0 0 0 1 0 5 0 0 0
C 30 4 130 5 1 0 0 0 0 0 1 2 4 3 1 0 0 0
D 0 0 0 148 14 49 2 4 7 12 17 17 1 0 1 29 1 4
E 0 0 0 2 167 3 12 7 16 2 3 6 2 1 0 0 1 0
F 0 0 0 40 5 151 1 7 12 31 17 27 0 0 0 25 13 0
G 0 0 0 1 5 0 98 3 19 3 3 10 0 0 1 2 22 3
H 0 0 0 1 3 0 15 84 5 30 8 23 0 0 0 0 0 0
I 0 0 0 7 9 3 32 6 66 34 23 14 0 1 0 2 16 5
J 0 0 0 5 2 1 17 73 51 69 45 25 1 0 0 24 7 1
K 0 0 1 6 11 1 19 9 4 11 81 16 0 2 0 5 2 0
L 0 0 0 10 12 6 46 30 46 37 44 67 0 0 0 2 2 3
M 95 1 80 2 4 1 3 1 0 1 1 8 236 9 3 5 5 2
O 0 2 4 0 0 0 0 0 0 0 0 0 2 164 40 1 0 2
P 0 7 0 0 1 0 0 0 0 0 0 0 2 30 190 3 8 2
Q 0 0 0 24 8 23 0 20 6 8 11 3 0 0 0 148 2 1
R 0 0 0 1 0 0 3 1 0 0 0 1 0 0 0 0 138 2
S 2 0 0 3 12 0 7 10 6 17 1 2 4 45 11 9 38 230
SVM Test Confusion
A B C D E F G H I J K L M O P Q R S
A 147 1 10 2 0 0 0 0 0 0 0 4 0 0 2 0 0 0
B 0 252 3 0 0 2 0 0 0 0 1 0 5 9 18 0 0 0
C 68 2 193 5 0 1 0 0 0 0 1 3 8 2 2 1 0 1
D 0 0 0 191 15 58 3 13 33 26 17 33 2 0 1 37 17 4
E 0 0 0 2 200 4 15 10 29 14 4 6 4 1 0 0 0 5
F 0 0 0 13 1 154 1 2 3 24 10 4 0 0 0 3 0 0
G 0 0 0 0 3 0 187 3 19 4 4 1 2 2 1 0 36 3
H 0 0 0 3 3 4 10 116 0 39 2 9 0 0 0 0 1 0
I 0 0 0 7 3 1 7 31 99 22 21 43 1 0 0 12 4 3
J 0 0 0 7 3 0 0 37 6 96 7 14 0 0 0 3 4 0
K 0 0 1 4 7 1 10 18 11 5 153 27 0 0 1 3 10 5
L 0 0 0 6 3 0 16 13 34 15 21 73 0 0 0 0 0 0
M 39 0 24 0 2 0 0 0 0 0 1 2 231 7 0 0 0 0
O 0 0 7 0 0 0 0 0 0 0 0 0 1 188 45 2 0 6
P 0 0 0 0 0 0 0 0 0 0 0 1 0 21 182 1 8 14
Q 0 0 0 12 7 13 0 11 4 8 13 1 0 0 0 186 1 0
R 0 0 0 0 0 0 3 0 0 1 0 0 0 1 0 0 173 1
S 1 0 0 3 8 0 3 1 0 1 0 0 1 24 3 7 1 213
RF Test Confusion
A B C D E F G H I J K L M O P Q R S
A 145 1 11 1 0 0 0 0 0 0 0 0 2 0 1 0 0 0
B 1 251 3 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0
C 41 1 187 4 0 0 1 0 1 0 1 1 16 1 4 0 0 2
D 2 0 0 164 10 51 2 5 29 22 18 40 0 0 0 22 13 6
E 0 0 0 21 193 1 22 2 7 1 0 5 9 1 0 0 1 4
F 0 0 0 14 8 158 1 5 1 23 8 4 0 0 0 0 5 0
G 0 0 0 0 2 0 178 14 3 1 0 4 5 5 4 0 15 2
H 0 0 0 1 4 16 1 120 0 14 0 0 0 0 0 1 0 1
I 0 0 0 9 3 3 4 27 136 5 8 20 0 0 0 1 1 6
J 0 0 1 4 3 1 15 39 9 151 5 24 0 1 0 5 11 1
K 0 0 0 12 3 1 1 16 15 6 167 24 0 1 3 4 5 1
L 0 0 0 9 16 0 20 17 35 26 43 89 0 0 0 10 0 2
M 52 0 31 0 1 1 2 1 0 0 1 1 210 6 3 0 0 1
O 13 2 5 0 0 0 6 0 0 0 0 0 7 180 45 1 0 2
P 1 0 0 0 0 0 0 0 0 0 0 5 0 29 172 3 6 0
Q 0 0 0 11 7 6 0 8 2 4 3 3 0 0 0 200 1 0
R 0 0 0 0 0 0 2 0 0 1 0 0 0 2 15 2 196 4
S 0 0 0 5 5 0 0 1 0 1 1 1 1 29 8 6 1 223

4.2.2 Binning Vote Matrices (First 10)

KNN Binned Predictions
User Action A B C D E F G H I J K L M O P Q R S WeightedPred
1 1606 A 4 0 4 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 M
18 1606 B 0 16 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 B
35 1606 C 0 1 14 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 C
52 1606 D 0 0 0 16 0 1 0 0 0 0 0 0 0 0 0 0 0 0 D
69 1606 E 0 0 0 0 15 0 0 0 0 0 0 0 0 0 1 0 0 1 E
86 1606 F 0 0 0 0 0 16 0 0 0 0 0 1 0 0 0 0 0 0 F
103 1606 G 0 0 0 0 0 0 3 0 0 0 0 6 0 0 0 0 2 6 L
120 1606 H 0 0 0 2 0 0 0 10 0 0 1 1 0 0 0 2 0 1 H
137 1606 I 0 0 0 0 0 1 1 2 1 1 0 11 0 0 0 0 0 0 L
154 1606 J 0 0 0 0 0 0 0 2 0 6 7 0 0 0 0 0 0 2 K
SVM Binned Predictions
User Action A B C D E F G H I J K L M O P Q R S WeightedPred
1 1606 A 6 0 10 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 C
18 1606 B 0 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 B
35 1606 C 0 0 16 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 C
52 1606 D 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D
69 1606 E 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 2 E
86 1606 F 0 0 0 0 0 16 0 0 1 0 0 0 0 0 0 0 0 0 F
103 1606 G 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 1 2 G
120 1606 H 0 0 0 0 0 0 0 14 0 3 0 0 0 0 0 0 0 0 H
137 1606 I 0 0 0 0 0 0 0 0 0 1 1 15 0 0 0 0 0 0 L
154 1606 J 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 H
RF Binned Predictions
User Action A B C D E F G H I J K L M O P Q R S WeightedPred
1 1606 A 12 0 4 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 A
18 1606 B 0 16 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 B
35 1606 C 1 1 12 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 C
52 1606 D 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D
69 1606 E 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 2 E
86 1606 F 0 0 0 0 0 16 0 0 1 0 0 0 0 0 0 0 0 0 F
103 1606 G 0 0 0 0 0 0 11 0 0 0 0 0 0 6 0 0 0 0 G
120 1606 H 0 0 0 0 0 0 2 9 0 4 0 2 0 0 0 0 0 0 H
137 1606 I 0 0 0 0 0 0 0 0 4 2 0 11 0 0 0 0 0 0 L
154 1606 J 0 0 0 0 0 0 1 1 0 14 0 1 0 0 0 0 0 0 J

4.2.3 Binning Confusion Matrices

KNN Binned Confusion
SVM Binned Confusion
A B C D E F G H I J K L M O P Q R S
A 9 0 4 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0
B 0 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
C 0 0 13 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
D 0 0 0 12 0 0 0 0 0 1 0 1 0 0 0 1 0 0
E 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 1 0 0
F 0 0 0 4 0 9 0 0 0 0 0 0 0 0 0 1 0 0
G 0 0 0 0 1 0 12 1 0 0 0 1 0 0 0 0 0 0
H 0 0 0 1 1 0 0 8 1 3 1 0 0 0 0 0 0 0
I 0 0 0 2 2 0 1 0 6 0 1 2 0 0 0 0 0 0
J 0 0 0 1 1 1 0 2 2 7 0 0 0 0 0 1 0 0
K 0 0 0 1 0 1 0 0 0 0 12 0 0 0 0 1 0 0
L 0 0 0 2 0 0 0 1 2 1 2 5 0 0 0 0 0 0
M 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0
O 0 1 0 0 0 0 0 0 0 0 0 0 0 11 2 0 0 1
P 0 1 0 0 0 0 0 0 0 0 0 0 0 3 11 0 0 0
Q 0 0 0 3 0 0 0 0 1 0 0 0 0 0 0 11 0 0
R 0 0 0 1 0 0 2 0 0 0 1 0 0 0 0 0 11 0
S 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 14
RF Binned Confusion
A B C D E F G H I J K L M O P Q R S
A 9 0 1 0 0 0 0 0 0 0 0 0 4 1 0 0 0 0
B 0 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
C 0 0 13 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
D 0 0 0 11 1 0 0 0 0 0 2 1 0 0 0 0 0 0
E 0 0 0 0 12 1 0 0 0 0 0 1 0 0 0 1 0 0
F 0 0 0 2 0 11 0 1 0 0 0 0 0 0 0 0 0 0
G 0 0 0 0 1 0 12 0 0 1 0 1 0 0 0 0 0 0
H 0 0 0 0 0 0 1 9 1 1 1 1 0 0 0 1 0 0
I 0 0 0 2 1 0 0 0 8 0 1 2 0 0 0 0 0 0
J 0 0 0 1 0 2 0 0 0 10 0 2 0 0 0 0 0 0
K 0 0 0 0 0 0 0 0 0 0 12 3 0 0 0 0 0 0
L 0 0 0 3 0 0 0 0 1 1 1 7 0 0 0 0 0 0
M 0 0 1 0 1 0 0 0 0 0 0 0 13 0 0 0 0 0
O 0 0 0 0 0 0 0 0 0 0 0 0 0 11 1 0 0 3
P 0 0 0 0 0 0 0 0 0 0 0 0 0 3 11 0 1 0
Q 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 12 0 0
R 0 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 12 0
S 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 14

5 Discussion

Within the discussion section we will converse about what steps could be done moving forward, issues met, and other auxiliary conversations that didn’t merit a place in the rest of the text.

5.1 Possibilities of Induced Error

What I observed to be one of the primary sources of error in the model was the lack of consistency in testing procedure that made it difficult to look at data across similar orientations. As an example, consider the plot below for action INSERT ACTION HERE and see how the mean of the acceleration varies from user to user.

5.2 Next Steps

If I had more time to work on this, here is what I have in mind to push this forward.

  • Revisit feature generation, especially in developing orientation agnostic features to mitigate the orientation inconsistency issue.
  • Open the model space up to neural networks, especially CNN with LSTM.

5.3 Open Issues

The primary open issue is in regard to the cross-validation results when compared to unseen data predictions. The cross validation should give us a good idea of the model accuracy and for all models the reported cross-validation accuracy is >90% across all folds whereas the predictions percentages are significantly lower. Out of curiosity, I ran k-folds for the entire data set, as opposed to just the training data, and the accuracy was very similar. This indicated to me that there may be something wrong with the predict method usage as a k-fold across the entire data set emulates 5 different test train splits. They should be similar but they are not.

6 Conclusion

Human activity recognition is an interesting and fruitful problem in the machine learning field. That being said, some limitations in the data and in open questions regarding the modeling have left us with a less than ideal prediction accuracy. Even so, the accuracy is nothing to be dissapointed with as we’re beating random selection classification by more than an order of magnitude. Of particular challenge was classifying the different eating activities as the nuances of the actions made it so macroscopic features could not consistently classify them.

7 Acknowledgements

I want to thank Dr. Donald Dabdub for creating this incredible course and opening up our eyes to the world of machine learning. Your course demystified the study and now I feel as though I have a grasp. This is now a field I am deeply considering pursuing further.

8 Appendix

Additional information that was excluded for the sake of brevity or lack of importance is here.

  1. Link to GitHub page for project (including prior revisions)
  2. Great page for starting with machine learning in R and Python

9 References

[1] G. Weiss, UCI Machine Learning Repository: WISDM Smartphone and Smartwatch Activity and Biometrics Dataset Data Set..

[2] Emile, “Features for time series classification,” Cross Validated..

[3] J. Matejka and G. Fitzmaurice, “The datasaurus dozen - same stats, different graphs,” Autodesk Research..

[4] Ataspinar, “Machine learning with signal processing techniques,” Ahmet Taspinar. Apr-2018.

[5] S. Rosati, G. Balestra, and M. Knaflitz, “Comparison of different sets of features for human activity recognition by wearable sensors,” Sensors (Basel, Switzerland). MDPI, Nov-2018.

[6] J. Brownlee, “Feature selection with the caret r package,” Machine Learning Mastery. Aug-2019.

[7] O. Banos, M. Damas, H. Pomares, and I. Rojas, “On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition,” Sensors (Basel, Switzerland). Molecular Diversity Preservation International (MDPI), 2012.

[8] J. Brownlee, “Your first machine learning project in r step-by-step,” Machine Learning Mastery. Oct-2019.

[9] “Sign in,” RPubs..

A work by Patrick Youssef

Patrick.S.Youssef@gmail.com